Image enhancement method and apparatus, and terminal device

ABSTRACT

Disclosed by the present application are an image enhancement method and apparatus, a terminal device and a computer-readable storage medium. The image enhancement method comprises: obtaining an image to be processed; performing a wavelet transform operation on the image to obtain raw feature information of the image, the raw feature information comprising global contour feature information, transversal detail feature information, longitudinal detail feature information, and contrast detail feature information; inputting the raw feature information into a trained target network for processing to obtain corresponding reconstruction feature information, the reconstruction feature information comprising global contour reconstruction information, transversal detail reconstruction information, longitudinal detail reconstruction information, and contrast detail reconstruction information; performing an inverse wavelet transform operation on the reconstruction feature information to obtain a reconstructed image; the resolution of the reconstructed image is higher than the resolution of the image to be processed.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a National Stage of PCT Application No.PCT/CN2019/120685 filed on Nov. 25, 2019, the content of which isincorporated herein by reference thereto.

TECHNICAL FIELD

The present application involves in the field of image processingtechnologies, and particularly relates to an image enhancement method,an image enhancement apparatus and a terminal device.

BACKGROUND

With the development of image processing technologies, image analysisand recognition are widely used in fields such as medical imaging, videomonitoring, remote sensing imaging, face recognition etc. In a specificapplication, due to limitations of a camera environment or a cameradevice, collected images are usually low-resolution images, and thelow-resolution images are required to be converted into high-resolutionimages to further analyze and recognize the images.

However, in the prior art, the high-resolution images obtained byconverting from the low-resolution images are relatively fuzzy, whichaffects the accuracy of subsequent image analysis and recognition.

SUMMARY

In view of this, embodiments of the present application provide an imageenhancement method, an image enhancement apparatus and a terminal deviceto solve the problem that the high-resolution images obtained byconverting from the low-resolution images are relatively fuzzy in theprior art.

A first aspect of the present application provides an image enhancementmethod, which includes:

acquiring an image to be processed;

performing a wavelet transform operation on the image to be processed toacquire raw feature information of the image to be processed, whereinthe raw feature information includes global contour feature information,transversal detail feature information, longitudinal detail featureinformation, and diagonal detail feature information;

inputting the raw feature information into a trained target network forprocessing to acquire corresponding reconstruction feature information;wherein the reconstruction feature information includes global contourreconstruction information, transversal detail reconstructioninformation, longitudinal detail reconstruction information and diagonaldetail reconstruction information, the target network is a generatorgroup acquired through training a first sample image and a correspondingsecond sample image based on four generative adversarial networks, andresolution of the first sample image is higher than resolution of thesecond sample image;

performing an inverse wavelet transform operation on the reconstructionfeature information to acquire a reconstructed image; where resolutionof the reconstructed image is higher than resolution of the image to beprocessed.

A second aspect of the present application provides an image enhancementapparatus, which includes:

a to-be-processed image acquisition unit configured to acquire an imageto be processed;

a wavelet transform unit configured to perform a wavelet transformoperation on the image to be processed to acquire raw featureinformation of the image to be processed, wherein the raw featureinformation includes global contour feature information, transversaldetail feature information, longitudinal detail feature information anddiagonal detail feature information;

-   -   a reconstruction feature information acquisition unit configured        to input the raw feature information into a trained target        network for processing to acquire corresponding reconstruction        feature information; wherein the reconstruction feature        information includes global contour reconstruction information,        transversal detail reconstruction information, longitudinal        detail reconstruction information and diagonal detail        reconstruction information, the target network is a generator        group acquired through training a first sample image and a        corresponding second sample image based on four generative        adversarial networks, and resolution of the first sample image        is higher than resolution of the second sample image;

an inverse wavelet transform unit configured to perform an inversewavelet transform operation on the reconstruction feature information toacquire a reconstructed image;

where resolution of the reconstructed image is higher than resolution ofthe image to be processed.

A third aspect of the present application provides a terminal device,which includes a memory, a processor and a computer program stored inthe memory and executable on the processor, and the processor, whenexecuting the computer program, implements the above-mentioned imageenhancement method.

A fourth aspect of the present application provides a computer-readablestorage medium, on which a computer program is stored, and the computerprogram, when executed by a processor, implements the above-mentionedimage enhancement method.

A fifth aspect of the present application provides a computer programproduct, and the computer program product, when executed on a terminaldevice, causes the terminal device to implement the image enhancementmethod described in the first aspect.

Beneficial Effect

In the embodiments of the present application, the raw featureinformation including the global contour feature information,transversal detail feature information, longitudinal detail featureinformation and diagonal detail feature information is obtained byperforming the wavelet transform operation on the image to be processed,and the reconstruction feature information including the global contourreconstruction information, transversal detail reconstructioninformation, longitudinal detail reconstruction information and diagonaldetail reconstruction information is acquired through the trained targetnetwork respectively, and then the inverse wavelet transform isperformed on the reconstruction feature information to acquire thereconstructed image having higher resolution than the resolution of theimage to be processed. Since the target network is a generator groupacquired through training the first sample image and the correspondingsecond sample image based on four generative adversarial networks, afterthe global contour feature information and the detail featureinformation in all directions of the image to be processed aredistinguished, the reconstruction feature information including theglobal contour reconstruction information, transversal detailreconstruction information, longitudinal detail reconstructioninformation and diagonal detail feature information can be generatedcorrespondingly and accurately through the target network, and then theinverse wavelet transform is performed to make every detail informationbe reconstructed accurately and separately, therefore the finalreconstructed image can be made clearer and more accurate.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly describe the technical solutions in theembodiments of the present application or in the prior art, drawingsthat need to be used in the description for the embodiments or the priorart will be briefly introduced below, it is obvious that the drawings inthe following description are only some embodiments of the presentapplication, and other drawings may be obtained by those of ordinaryskill in the art based on these drawings without creative work.

FIG. 1 is a schematic diagram of an implementation process of a firstimage enhancement method provided by the present application;

FIG. 2 is a schematic diagram of a flowchart of an image enhancementmethod provided by the present application;

FIG. 3 is a schematic diagram of a wavelet transform operation providedby the present application;

FIG. 4 is a schematic diagram of a system architecture of an imageenhancement method provided by the present application;

FIG. 5 is a schematic diagram of a network structure of a generatorprovided by the present application;

FIG. 6 is a schematic diagram of an implementation process of a secondimage enhancement method provided by the present application;

FIG. 7 is a schematic diagram of an image enhancement apparatus providedby the present application;

FIG. 8 is a schematic structural diagram of an embodiment of a terminaldevice provided by the present application.

DETAILED DESCRIPTION

In the following description, for the purpose of illustration ratherthan limitation, specific details such as a specific system structureand technology etc. are proposed for a thorough understanding of theembodiments of the present application. However, it should be understoodto those skilled in the art that the present application can also beimplemented in other embodiments without these specific details. Inother cases, detailed descriptions of well-known systems, apparatuses,circuits, and methods are omitted to avoid unnecessary details fromobstructing the description of the present application.

Please refer to FIG. 1 , FIG. 1 is a schematic flowchart of a firstimage enhancement method provided by an embodiment of the presentapplication. An execution subject of the image enhancement method inthis embodiment is a terminal device, which includes but is not limitedto a mobile terminal such as a smart phone, a tablet computer, and a PDA(Personal Digital Assistant) etc., and may also include a terminaldevice such as a desktop computer and a server etc. The imageenhancement method as shown in FIG. 1 includes the following.

At S101: acquire an image to be processed.

The image to be processed is an image provided with a low resolution andrequiring image enhancement, the image to be processed may be acquiredby an image acquisition device, alternatively the image to be processedmay be acquired through reading from a storage unit of a local terminalor a third party. According to a specific application scenario of theimage enhancement method in an embodiment of the present application,the image to be processed may specifically be a low-resolution medicalimage, a low-resolution face image, a low-resolution remote sensingimage, and the like.

At S102: perform a wavelet transform operation on the image to beprocessed to acquire raw feature information of the image to beprocessed, where the raw feature information includes global contourfeature information, transversal detail feature information,longitudinal detail feature information, and diagonal detail featureinformation.

The image to be processed is input into a discrete wavelet transformmodel to perform the wavelet transform operation on the image to beprocessed, so as to acquire the raw feature information of the image tobe processed. Specifically, firstly spectrum information of the image tobe processed is acquired according to a discrete wavelet transformfunction and various frequency components in the spectrum informationare extracted; then the various frequency components are combined intofour frequency combination components, and the four frequencycombination components are separately converted into a spatial frequencydomain to acquire the corresponding global contour feature information,transversal detail feature information, longitudinal detail featureinformation and diagonal detail feature information, and these fourpieces of feature information are the raw feature information. As shownin FIG. 2 , the image to be processed is input into the DWT (DiscreteWavelet Transformation) model to acquire the corresponding raw featureinformation, and the four images from top to bottom of the raw featureinformation are the global contour feature information, the transversaldetail feature information, the longitudinal detail feature informationand the diagonal detail feature information respectively.

Further, the wavelet transform operation is specifically a compactlysupported orthogonal wavelet transform operation provided with symmetry.

Since the compactly supported orthogonal wavelet transform can speed upcalculation speed and save operation time, and the symmetrical andcompactly supported orthogonal wavelet transform can further effectivelyavoid phase distortion during image processing, thereby the extractionof the image feature information is more accurate. Specifically, anembodiment of the present application adopts an approximatelysymmetrical and compactly supported orthogonal wavelet transformfunction to perform wavelet transformation, and a mathematicalexpression of the approximately symmetrical and compactly supportedwavelet transform is as follows:

$\begin{matrix}{{x(t)} = {{\sum\limits_{k \in Z}{u_{j_{0},k}\phi_{{j0},{k(t)}}}} + {\sum\limits_{j = {- \infty}}^{j_{0}}{\sum\limits_{k \in Z}{u_{j_{0},k}\phi_{j_{0},{k(t)}}}}}}} & \lbrack 1\rbrack\end{matrix}$

where ϕ_(j,k)(t)=2^(j)ϕ(2^(j)t−k) is a scale function,ψ_(j,k)(t)=2^(j)ψ(2^(j)t−k) is a wavelet function; u_(j,k) is a scalecoefficient and is equal to inner product of x and ϕ_(j,k), i.e.,u_(j,k)=<X,ϕ_(j,k)>; ω_(j,k) is a wavelet coefficient and is equal toinner product of x and ψ_(j,k), i.e., ω_(j,k)=<x,ψ_(j,k)>; j₀ is anarbitrary value, which indicates an arbitrary starting scale.

Further, the image to be processed is specifically a three-dimensionalimage, and the performing the wavelet transform operation on the imageto be processed to acquire the raw feature information of the image tobe processed includes:

at S10201: performing a wavelet transform operation on the image to beprocessed in an x-axis direction to acquire first spectrum information;

at S10202: performing a wavelet transform operation on the firstspectrum information in a y-axis direction to acquire second spectruminformation;

at S10203: performing a wavelet transform operation on the secondspectrum information in a z-axis direction to acquire third spectruminformation;

at S10204: acquiring the raw feature information according to the thirdspectrum information.

The image to be processed in this embodiment of the present applicationis specifically a three-dimensional image, so it is necessary to performa three-dimensional wavelet transform operation on this image to beprocessed, that is, three wavelet transform operations including thewavelet transform in the x-axis direction, the wavelet transform in they-axis direction and the wavelet transform in the z-axis direction areperformed in sequence to extract the frequency component information ofthe image to be processed in each direction, and then the frequencycomponent information in each direction is converted into the spatialfrequency domain to acquire the raw feature information.

Specifically, the schematic diagram of the three wavelet transformoperations is shown in FIG. 3 , where “⬇x2” indicates a down-samplingoperation with a sampling interval of 2 in the x-axis direction, “⬇y2”indicates a down-sampling operation with a sampling interval of 2 in they-axis direction, and “⬇z2” indicates a down-sampling operation with asampling interval of 2 in the z-axis direction, which are detailedbelow.

In the S10201, the wavelet transform operation is performed on the imageto be processed in the x-axis direction to acquire the first spectruminformation, and the first spectrum information includes a firstfrequency component and a second frequency component, where the firstfrequency component includes low-frequency component information in thex-axis direction, and the second frequency component includeshigh-frequency component information in the x-axis direction.

In the S10202, after the first spectrum information is down sampled, thewavelet transform operation is performed in the y-axis direction toacquire the second spectrum information, and the second spectruminformation includes a third frequency component, a fourth frequencycomponent, a fifth frequency component and the sixth frequencycomponent. Specifically, the third frequency component includeslow-frequency component information in the x-axis direction andlow-frequency component information in the y-axis direction; the fourthfrequency component includes low-frequency component information in thex-axis direction and high-frequency component information in the y-axisdirection; the fifth frequency component includes high-frequencycomponent information in the x-axis direction and low-frequencycomponent information in the y-axis direction; and the sixth frequencycomponent includes high-frequency component information in the x-axisdirection and high-frequency component information in the y-axisdirection.

In the S10203, after the second spectrum information is down sampled,the wavelet transform operation is performed in the z-axis direction toacquire the third spectrum information, and the third spectruminformation includes eight frequency components.

In the S10204, after the eight frequency components of the thirdspectrum information are down sampled respectively, eight targetcomponents are acquired. Specifically, a first target component includeslow-frequency component information in the x-axis direction,low-frequency component information in the y-axis direction andlow-frequency component information in the z-axis direction; a secondtarget component includes low-frequency component information in thex-axis direction, low-frequency component information in the y-axisdirection and high-frequency component information in the z-axisdirection; a third target component includes low-frequency componentinformation in the x-axis direction, high-frequency componentinformation in the y-axis direction and low-frequency componentinformation in the z-axis direction; a fourth target component includeslow-frequency component information in the x-axis direction,high-frequency component information in the y-axis direction andhigh-frequency component information in the z-axis direction; a fifthtarget component includes high-frequency component information in thex-axis direction, low-frequency component information in the y-axisdirection and low-frequency component information in the z-axisdirection; a sixth target component includes high-frequency componentinformation in the x-axis direction, low-frequency component informationin the y-axis direction and high-frequency component information in thez-axis direction; a seventh target component includes high-frequencycomponent information in the x-axis direction, high-frequency componentinformation in the y-axis direction and low-frequency componentinformation in the z-axis direction; and an eighth target componentincludes high-frequency component information in the x-axis direction,high-frequency component information in the y-axis direction andhigh-frequency component information in the z-axis direction.

Afterwards, the spatial frequency domain transform is performed based ona combination of the first target component and the second targetcomponent to acquire the global contour feature information; the spatialfrequency domain transform is performed based on a combination of thethird target component and the fourth target component to acquire thetransversal detail feature information; the spatial frequency domaintransform is performed based on a combination of the fifth targetcomponent and the sixth target component to acquire the longitudinaldetail feature information; and the spatial frequency domain transformis performed based on a combination of the seventh target component andthe eighth target component to acquire the diagonal detail featureinformation, thereby acquiring the complete raw feature information.

In the embodiment of the present application, when the image to beprocessed is a three-dimensional image, three transformations arerespectively performed from three directions including the x axis, yaxis and z axis to accurately acquire each frequency component, and thenthe frequency components are combined and transformed to acquire thecorresponding global contour feature information, transversal detailfeature information, longitudinal detail feature information anddiagonal detail feature information, which can make the extraction ofthe raw feature information more complete and accurate.

At S103: input the raw feature information into a trained target networkfor processing to acquire corresponding reconstruction featureinformation; where the reconstruction feature information includesglobal contour reconstruction information, transversal detailreconstruction information, longitudinal detail reconstructioninformation and diagonal detail reconstruction information; the targetnetwork is a generator group acquired through training a first sampleimage and a corresponding second sample image based on four generativeadversarial networks; resolution of the first sample image is higherthan resolution of the second sample image.

As shown in FIG. 2 , the raw feature information is input into thetrained target network for processing to acquire the correspondingreconstruction feature information, and the reconstruction featureinformation includes the global contour reconstruction information, thetransversal detail reconstruction information, the longitudinal detailreconstruction information and the diagonal detail reconstructioninformation, which respectively correspond to the four images from topto bottom in the reconstruction feature information of FIG. 2 .Specifically, the target network is a generator group containing fourgenerators. Specifically, in a system architecture as shown in FIG. 4 ,the target network is a generator group acquired through training thefirst sample image and the corresponding second sample image based onfour generative adversarial networks; here the resolution of the firstsample image is higher than the resolution of the second sample image.Specifically, the four generative adversarial networks are composed of agenerator group and a discriminator group, the generator group includesa first generator G_(A), a second generator G_(H), a third generatorG_(V) and a fourth generator G_(D), and the discriminator group includesa first discriminator D_(A), a second discriminator D_(H), a thirddiscriminator D_(V) and a fourth discriminator D_(D); the firstgenerator G_(A) corresponds to the first discriminator D_(A) to form afirst generative adversarial network, the second generator G_(H)corresponds to the second discriminator D_(H) to form a secondgenerative adversarial network, the third generator G_(V) corresponds tothe third discriminator D_(V) to form a third generative adversarialnetwork, and the fourth generator G_(D) corresponds to the fourthdiscriminator D_(D) to form a fourth generative adversarial network.

Further, the four generative adversarial networks are specifically fourrelative average generative adversarial networks. Since the relativeaverage generative adversarial network, during training, constrains thediscriminator and generator based on relative discriminant probability,and comprehensively considers relative authenticity probability betweena real sample and generated data, so as to solve the problem ofinstability in the training process of the generative adversarialnetwork, thereby achieving more accurate training, and making theextraction of detailed features by the final acquired target networkmore accurate.

Further, the trained target network includes a first generator, a secondgenerator, a third generator and a fourth generator, and the step S103includes:

at S10301: inputting the global contour feature information into thefirst generator for processing to acquire the global contourreconstruction information;

at S10302: inputting the transversal detail feature information into thesecond generator for processing to acquire the transversal detailreconstruction information;

at S10303: input the longitudinal detail feature information into thethird generator for processing to acquire the longitudinal detailreconstruction information;

at S10304: inputting the diagonal detail feature information into thefourth generator for processing to acquire the diagonal detailreconstruction information.

In an embodiment of the present application, the target networkspecifically includes a first generator G_(A), a second generator G_(H),a third generator G_(V) and a fourth generator G_(D).

In the S10301, the global contour feature information in the raw featureinformation is specifically input into the first generator G_(A) tolearn and reconstruct the global contour features, so as to acquire thecorresponding global contour reconstruction information.

In the S10302, the transversal detail feature information in the rawfeature information is specifically input into the second generatorG_(H) to learn and reconstruct the transversal detail features, so as toacquire the corresponding transversal detail reconstruction information.

In the S10303, the longitudinal detail feature information in the rawfeature information is specifically input into the third generator G_(V)to learn and reconstruct the transversal detail features, so as toacquire the corresponding longitudinal detail reconstructioninformation.

In the S10304, the diagonal detail feature information in the rawfeature information is specifically input into the fourth generatorG_(D) to learn and reconstruct the transversal detail features toacquire the corresponding diagonal detail reconstruction information.

It should be understood that the above steps from S10301 to S10304 areperformed independently, and these four steps may be performedsimultaneously or sequentially in any order. Further, these four stepsare specifically performed at the same time, thereby improving theprocessing efficiency of the target network.

In the embodiments of the present application, the four generators ofthe target network respectively accurately reconstruct the four piecesof feature information to acquire the four pieces of reconstructioninformation, so that the target network extracts the detail featuresmore accurately.

Further, the first generator, the second generator, the third generatorand the fourth generator are all residual network structures including nresidual blocks, where n is a positive integer; correspondingly, theinputting the raw feature information into the trained target networkfor processing to acquire the corresponding reconstruction featureinformation includes:

at S10301A: inputting the global contour feature information into thefirst generator, and acquiring n first intermediate feature informationthrough n first residual blocks;

and acquiring the global contour reconstruction information according tothe n first intermediate feature information;

at 510302A: inputting the transversal detail feature information intothe second generator, and acquiring n second intermediate featureinformation through n second residual blocks; and acquiring thetransversal detail reconstruction information according to the n secondintermediate feature information;

at 510303A: inputting the longitudinal detail feature information intothe third generator, and acquiring n third intermediate featureinformation through n third residual blocks; and acquiring thelongitudinal detail reconstruction information according to the n thirdintermediate feature information;

at 510304A: inputting the diagonal detail feature information into thefourth generator, and acquiring n fourth intermediate featureinformation through n fourth residual blocks; and acquiring the diagonaldetail reconstruction information according to the n fourth intermediatefeature information.

In the embodiments of the present application, each generator in thetarget network is a residual network structure including n residualblocks, where n is a positive integer.

As shown in FIG. 5 , the network structure of each generatorspecifically includes a first 3D convolutional layer, n residual blocks,(n−1) long connections (where each long connection contains a second 3Dconvolutional layer), an up-sampling layer and a third 3D convolutionallayer. Among them, each residual block includes two 3D convolutionallayers and an activation function between the two 3D convolutionallayers.

Specifically, as an example, n is 20, the first 3D convolutional layeris a 9×9×9 convolutional layer, each residual block consists of a 3×3×3convolutional layer, a Leaky ReLU activation function and a 3×3×3convolutional layer, the second 3D convolutional layer is a 1×1×1convolutional layer, and the third 3D convolutional layer is a 3×3×3convolutional layer.

Specifically, the step S10301A is taken as an example, the details ofwhich are as follows.

At S10301A1: input the global contour feature information into the first3D convolution of the first generator to acquire a first feature vector.

At S10301A2: input the first feature vector into the n first residualblocks in sequence to acquire the n first intermediate featureinformation. Specifically, after data processing is performed on eachresidual block in the preceding (n−1) first residual blocks, thegenerated feature information is input into the next residual block forprocessing and at the same time is connected to the end of the lastfirst residual block as intermediate feature information through onelong connection including the second 3D convolutional layer; theinformation output by the last first residual block is also regarded asintermediate feature information, which combines other (n−1) pieces ofintermediate feature information acquired through long connections toacquire n pieces of intermediate feature information for weightingprocessing, so as to acquire a second feature vector.

At S10301A3: input the second feature vector into the up-sampling layerfor up-sampling processing to acquire a third feature vector.

At S10301A4: input the third feature vector into the last third 3Dconvolutional layer to acquire the global contour reconstructioninformation.

The processing processes of the generators in the steps from S10302A toS10304A for the detail feature information are similar to the processingprocess of the step S10301A, which will not be repeated herein again.

In the embodiments of the present application, since each generator is aresidual network structure, the intermediate feature information may bedirectly connected to the last residual block through the longconnection, so that the subsequent network layer can directly learn theprevious intermediate feature information, and the problem ofinformation loss and depletion when the feature information istransmitted through the convolutional layers is reduced, therebyimproving the accuracy of feature extraction.

At S104: perform an inverse wavelet transform operation on thereconstruction feature information to acquire a reconstructed image;here resolution of the reconstructed image is higher than resolution ofthe image to be processed.

As shown in FIG. 2 , the reconstruction feature information obtainedthrough processing by the target network is input into an IWT (InverseWavelet Transformation) model to form the inverse wavelet transformationoperation, the various pieces of detail feature information aresynthesized and transformed to acquire the reconstructed image, and theresolution of the reconstructed image is higher than the resolution ofthe image to be processed. Since the global contour reconstructioninformation, the transversal detail reconstruction information, thelongitudinal detail reconstruction information and the diagonal detailreconstruction feature information in the reconstruction featureinformation are information obtained through the respective accuratereconstruction by the trained target network, therefore loss of detailedfeature information can be avoided, and the clarity and accuracy of thefinal synthesized reconstructed image can be improved.

In the embodiments of the present application, the raw featureinformation including the global contour feature information,transversal detail feature information, longitudinal detail featureinformation and diagonal detail feature information is obtained byperforming the wavelet transform operation on the image to be processed,and the reconstruction feature information including the global contourreconstruction information, transversal detail reconstructioninformation, longitudinal detail reconstruction information and diagonaldetail reconstruction information is acquired through the trained targetnetwork respectively, and then the inverse wavelet transform isperformed on the reconstruction feature information to acquire thereconstructed image having a higher resolution than the resolution ofthe image to be processed. Since the target network is a generator groupacquired through training the first sample image and the correspondingsecond sample image based on four generative adversarial networks, afterthe global contour feature information and the detailed featureinformation in all directions of the image to be processed aredistinguished, the reconstruction feature information including theglobal contour reconstruction information, transversal detailreconstruction information, longitudinal detail reconstructioninformation and diagonal detail feature information can be generatedcorrespondingly and accurately through the target network, and then theinverse wavelet transform is performed to make every detail informationbe reconstructed accurately and separately, therefore the finalreconstructed image can be made clearer and more accurate.

Please refer to FIG. 6 , FIG. 6 is a schematic flowchart of a secondimage enhancement method provided by an embodiment of the presentapplication. An execution subject of the image enhancement method inthis embodiment is a terminal device, which includes but is not limitedto a mobile terminal such as a smart phone, a tablet computer, and a PDA(Personal Digital Assistant) etc., and may also include a terminaldevice such as a desktop computer and a server etc.

This embodiment adds training steps from S601 to S602 of the targetnetwork on the basis of the previous embodiment. the steps from S603 toS606 in this embodiment are exactly the same as the steps from S101 toS104 in the previous embodiment, the details refer to the relevantdescription of the steps from S101 to S104 in the previous embodiment,which will not be repeated herein again. In the image enhancement methodas shown in FIG. 6 , in order to improve the accuracy of thereconstruction feature information acquired by the target network, theS601 and S602 are as follows.

At S601: acquire a first sample image, and perform down-sampling on thefirst sample image to acquire a corresponding second sample image.

The first sample image having a higher resolution is acquired, and adown-sampling operation is performed on the first sample image toacquire the corresponding second sample image having a lower resolution.Each first sample image corresponds to one second sample image acquiredthrough down-sampling.

At S602: train four generative adversarial networks according to thefirst sample image and the second sample image to acquire the trainedtarget network, where the four generative adversarial networks include agenerator group and a corresponding discriminator group, and the targetnetwork is the generator group in the four generative adversarialnetworks.

As shown in FIG. 4 , the four generative adversarial networks in anembodiment of the present application include the generator group andthe discriminator group, the generator group includes a first generatorG_(A), a second generator G_(H), a third generator G_(V) and a fourthgenerator G_(D), and the discriminator group includes a firstdiscriminator D_(A), a second discriminator D_(H), a third discriminatorD_(V) and a fourth discriminator D_(D); the first generator G_(A)corresponds to the first discriminator D_(A) to form the firstgenerative adversarial network, the second generator G_(H) correspondsto the second discriminator D_(H) to form a second generativeadversarial network, the third generator G_(V) corresponds to the thirddiscriminator D_(V) to form the third generative adversarial network,and the fourth generator G_(D) corresponds to the fourth discriminatorD_(D). The target network in the embodiment of the present applicationis specifically the generator group in the four generative adversarialnetworks.

The four generative adversarial networks are trained according to thefirst sample image having higher resolution and the second sample imagehaving lower resolution and corresponding to the first sample image toacquire four trained generative adversarial networks. The generatorgroup is acquired from the four trained generative adversarial networksto acquire the trained target network.

Further, the step S602 includes:

at S60201: performing wavelet transformation on the first sample imageto acquire first raw feature information of the first sample image,where the first raw feature information includes first global contourfeature information, first transversal detail feature information, firstlongitudinal detail feature information and first diagonal detailfeature information;

at S60202: performing wavelet transform on the second sample image toacquire second raw feature information of the second sample image, wherethe second raw feature information includes second global contourfeature information, second transversal detail feature information,second longitudinal detail feature information and second diagonaldetail feature information;

at S60203: inputting the second raw feature information into thegenerator group for processing to acquire corresponding sample datareconstruction information, where the sample data reconstructioninformation includes global sample contour reconstruction information,transversal sample detail reconstruction information, longitudinalsample detail reconstruction information and diagonal sample detailreconstruction information;

at S60204: inputting the first raw feature information and the sampledata reconstruction information into the discriminator group forprocessing to acquire a corresponding discrimination output result;

at S60205: calculating loss values of the discriminators in thediscriminator group and loss values of the generators in the generatorgroup according to the discrimination output result;

at S60206: iteratively updating network parameters of the discriminatorsand the generators through a gradient descent backpropagation algorithmrespectively according to each of the loss values of the discriminatorsand each of the loss values of the generators, and minimizing each ofthe loss values of the discriminators and each of the loss values of thegenerators to acquire the trained target network.

In the S60201, the first sample image is input into the DWT (DiscreteWavelet Transform) model to perform wavelet transform and transform intothe spatial frequency domain so as to acquire the first raw featureinformation of the first sample image, and the first raw featureinformation includes the first global contour feature information, thefirst transversal detail feature information, the first longitudinaldetail feature information and the first diagonal detail featureinformation that are four pieces of feature information throughextraction.

In the S60202, the second sample image is input into the DWT (DiscreteWavelet Transform) model to perform the wavelet transform and transforminto the spatial frequency domain so as to acquire the second rawfeature information of the second sample image, and the second rawfeature information includes the second global contour featureinformation, the second transversal detail feature information, thesecond longitudinal detail feature information and the second diagonaldetail feature information that are four pieces of feature informationthrough extraction.

In the S60203, the second raw feature information is input into thegenerator group for processing to acquire the corresponding sample datareconstruction information. Specifically, the second global contourfeature information in the second raw feature information is input intothe first generator in the generator group for processing to acquire theglobal sample contour reconstruction information in the sample datareconstruction information; the second transversal detail featureinformation is input into the second generator in the generator groupfor processing to acquire the corresponding transversal sample detailreconstruction information; the second longitudinal detail featureinformation is input into the third generator in the generator group forprocessing to acquire the corresponding longitudinal sample detailreconstruction information; and the second diagonal detail featureinformation is input into the fourth generator for processing to acquirethe corresponding diagonal sample detail reconstruction information.

In the S60204, specifically, the first global contour featureinformation in the first raw feature information and the global samplecontour reconstruction information yA in the sample data reconstructioninformation are input into the first discriminator in the discriminatorgroup for processing to acquire the discrimination output result of thefirst discriminator; the first transversal detail feature information inthe first raw feature information and the transversal sample detailreconstruction information in the sample data reconstruction informationare input into the second discriminator in the discriminator group forprocessing to acquire the discrimination output result of the seconddiscriminator; the first longitudinal detail feature information in thefirst raw feature information and the longitudinal sample detailreconstruction information in the sample data reconstruction informationare input into the third discriminator in the discriminator group forprocessing to acquire the third discrimination output result; and thefirst diagonal detail feature information in the first raw featureinformation and the diagonal sample detail reconstruction information inthe sample data reconstruction information are input into the fourthdiscriminator in the discriminator group for processing to acquire thefourth discrimination output result.

In the S60205, the corresponding loss values of the discriminators(which specifically includes the loss value LossD_(A) of the firstdiscriminator, the loss value LossD_(H) of the second discriminator, theloss value LossD_(V) of the third discriminator and the loss valueLossD_(D) of the fourth discriminator) are calculated according to thediscrimination output results and preset formulas for calculating theloss values of the discriminators. According to the discriminationoutput results and the preset formulas for calculating the loss valuesof the generators, the loss values of the generators acquired throughcalculation specifically includes the loss value LossG_(A) of the firstgenerator, the loss value LossG_(H) of the second generator, the lossvalue LossG_(V) of the third generator, and the loss value LossG_(D) ofthe fourth generator.

Further, the four generative adversarial networks in the embodiment ofthe present application are all relative mean generative adversarialnetworks, in the relative mean generative adversarial network,calculation formulas of its loss value Loss_(D) of the discriminator andits loss value Loss_(G) of the generator are calculated as follows:

[2]LossD=−E _(x) _(r) _(˜P)[log(D(x _(r) ,x _(f)))]−E _(x) _(f)_(˜Q)[log(1−(x _(f) ,x _(r)))]  (1)

[3]LossG=−E _(x) _(r) _(˜P)[log(1−D(x _(r) ,x _(f)))]−E _(x) _(f)_(˜Q)[log(D(x _(f) ,x _(r)))]  (1)

where, x_(r) represents real data directly input into the discriminator,and x_(f) data represents generated data input into the discriminatorafter generated by the generator; D(x_(r),x_(f))=sigmoid(C(x_(f))−E[C(x_(f))]), C(x_(r)) represents probability that thediscriminator discriminates the real data x_(r) as true, and C(x_(f))represents probability that the discriminator discriminates thegenerated data x_(f) as true.

According to the discrimination output results, based on the formula(1), the discriminator loss values corresponding to the discriminatorscan be calculated. specifically:

Regarding the loss value LossD_(A) of the first discriminator, let x_(r)in the formula (1) be specifically the first global contour featureinformation y_(A) of the first sample image, let x_(f) in the formula bespecifically the global sample contour reconstruction information x_(A)generated by the first generator, and D_(A)(y_(A), x_(A)) is determinedaccording to the output result of the first discriminator, to acquireLossD_(A)=−E_(y) _(A) _(˜P)[log(1−D_(A)(y_(A),x_(A)))]−E_(x) _(A)_(˜Q)[log(1−D_(A)(x_(A),y_(A)))]; similarly, regarding the loss valueLossG_(A) of the first generator, according to the formula (2), thereis: LossG_(A)=−E_(y) _(A) _(˜P)[Log(1−D_(A)(y_(A),x_(A)))]−E_(x) _(A)_(˜Q)[Log(D(x_(A), y_(A)))].

Regarding the loss value LossD_(H) of the second discriminator, letx_(r) in the formula (1) be specifically the first transversal detailfeature information y_(H) of the first sample image, let x_(f) in theformula be specifically the transversal sample detail reconstructioninformation x_(H) generated by the second generator, and D_(H)(y_(H),x_(H)) is determined according to the output result of the seconddiscriminator, to acquire LossD_(H)=−E_(y) _(H)_(˜P)[log(D_(H)(y_(H),x_(H)))]−E_(x) _(H) _(˜Q) [log(1−D_(H) (x_(H),y_(H)))]; similarly, regarding the loss value LossG_(H) of the secondgenerator, according to the formula (2), there is: LossG_(H)=−E_(y) _(H)_(˜P)[log(1−D_(H)(y_(H), x_(H)))]−E_(x) _(H) _(˜Q) [log(D(x_(H),y_(H)))].

Regarding the loss value LossD_(V) of the third discriminator, let x_(r)in the formula (1) be specifically the first longitudinal detail featureinformation y_(V) of the first sample image, let x_(f) in the formula bespecifically the longitudinal sample detail reconstruction informationxx generated by the third generator, and D_(V)(y_(V), x_(V)) isdetermined according to the output result of the third discriminator, toacquire LossD_(V)=−E_(y) _(V) _(˜P) [log(D_(V) (y_(H), x_(H)))]−E_(x)_(y) _(˜Q)[log(1−D_(V) (x_(V), y_(r)))]; similarly, regarding the lossvalue LossG_(V) of the third generator, according to the formula (2),there is: LossG_(V)=−E_(y) _(V) _(˜P)[log(1−D_(V)(y_(V), x_(V)))]−E_(x)_(V) _(˜Q)[log(D(x_(V), y_(V)))].

Regarding the loss value LossD_(D) of the fourth discriminator, letx_(r) in the formula (1) be specifically the first transversal detailfeature information y_(D) of the first sample image, let x_(f) in theformula be specifically the transversal sample detail reconstructioninformation x_(D) generated by the fourth generator, and D_(D)(y_(D),x_(D)) is determined according to the output result of the fourthdiscriminator, to acquire LossD_(D)=−E_(y) _(D)_(˜P)[log(D_(D)(y_(D),x_(D)))]−E_(x) _(D) _(˜Q)[log(1−D_(D)(x_(D),y_(D)))]; similarly, regarding the loss value LossG_(D) of the fourthgenerator, according to the formula (2), there is: LossG_(D)=−E_(y) _(D)_(˜P)[log(1−D_(V)(y_(D), x_(D)))]−E_(x) _(D) _(˜Q)[log(D(x_(D),y_(D)))].

In the embodiments of the present application, since the relative meangenerative adversarial network constrains the discriminators andgenerators during training based on the relative discriminationprobability, comprehensively considers the probability of the relativeauthenticity between the real sample and the generated data, so as tosolve the problem of instable training process of the generativeadversarial network, thereby achieving more accurate training and makingthe extraction of the detail features by the final acquired targetnetwork more accurate.

In the S60206, according to the loss values of the discriminators andthe loss values of the generators calculated in the step S60205, thegradient descent algorithm is used to calculate the parameter values ofthe discriminators and the generators required to be adjusted, and therespective network parameters of the discriminators and the generatorsare iteratively updated through back propagation, the loss values of thediscriminators and the loss values of the generators are minimized toacquire the four trained generative adversarial networks, and thetrained generator group is acquired therein as the trained targetnetwork. Specifically, each generative adversarial network isindependently trained, and the trained generative adversarial network isobtained when all the four generative adversarial networks complete thetraining. Specifically, when each generative adversarial network istrained, the network parameters of the generators are fixed first, theloss values of the discriminators are minimized, the network parametersof the discriminators are updated by backpropagation, to complete thetraining of the discriminators; afterward, the network parameters of thediscriminators are fixed, the loss values of the generators areminimized, the network parameters of the generators are updated throughbackpropagation, to complete the training of the generators, therebycompleting the training of the generative adversarial network.

Further, after the step S60203, the method further includes:

at S602031: performing an inverse wavelet transform operation on thesample data reconstruction information to acquire a reconstructed sampleimage;at S602032: comparing the reconstructed sample image with thecorresponding first sample image pixel by pixel, and calculatingpixel-by-pixel difference loss values;correspondingly, the step S60206 specifically includes:at S60206A: according to the loss values of the discriminators, the lossvalues of the generators, and the pixel-by-pixel difference loss values,iteratively updating the network parameters of the discriminators andthe generators through a gradient descent backpropagation algorithmrespectively, and minimizing the loss values of the discriminators, theloss values of the generators and the pixel-by-pixel difference lossvalues to acquire the trained target network.

In the embodiments of the present application, when the target networkis trained based on the four generative adversarial networks, inaddition to minimizing the loss values of the discriminators and theloss values of the generators, the pixel-by-pixel difference lossvalues, calculated by comparing the reconstructed sample image with thecorresponding first sample image pixel by pixel, is added on this basis,and the pixel-by-pixel difference loss values are further minimized tofurther improve the accuracy of the trained target network.

Specifically, in the S602031, the inverse wavelet transform is performedaccording to the global sample contour reconstruction information, thetransversal sample detail reconstruction information, the longitudinalsample detail reconstruction information, and the diagonal sample detailreconstruction information in the sample data reconstructioninformation, to synthesize the reconstructed sample image.

In the S602032, the reconstructed sample image and the correspondingfirst sample image (that is, the first sample image corresponding to thesecond sample image before reconstruction) are input into apixel-by-pixel comparison module, and each pixel information in the twoimages is compared one by one, and the pixel-by-pixel difference lossvalue LossF is calculated. Among them, LossF=E_(xe,X,yeY)[∥y−G(x)∥₁], yrepresents the real first sample image having higher resolution, G(x)represents the reconstructed sample image generated by the generator,and ∥ ∥₁ represents the L1 paradigm, here the use of the L1 paradigm isuseful to make the edges of the generated image clearer.

In the S60206A, specifically, when each generative adversarial networkis trained, the network parameters of the generators are fixed first,the loss values of the discriminators are minimized, the networkparameters of the discriminators are updated by backpropagation tocomplete the training of the discriminators; then the network parametersof the discriminators are fixed, the loss values of the generators areminimized, and the network parameters of the generators are updatedthrough backpropagation; afterward, the pixel-by-pixel difference lossvalue LossF is minimized, and the network parameters of the generatorsand the discriminators are further updated through backpropagation,thereby completing the training of the generative adversarial network.After the training of each generative adversarial network is completed,the four trained generative adversarial networks are obtained. The fourgenerators of the four generative adversarial networks are extracted toacquire the generator group as the trained target network.

Further, after the S60205, the method further includes:

at S602051: acquiring a loss value of the generative adversarial networkaccording to the discrimination loss function corresponding to each ofthe discriminators and a global contour weight, a texture detail weight,a transversal detail weight, a longitudinal detail weight and a diagonaldetail weight;at S602052: calculating a total loss value according to the loss valueof the generative adversarial network, the pixel-by-pixel differenceloss value, a loss weight of the generative adversarial network and apixel-by-pixel difference loss weight;correspondingly, the S60206 specifically includes:at 560206B: iteratively updating the respective network parameters ofthe discriminators and the discriminators through the gradient descentbackpropagation algorithm according to the loss values of thediscriminators, the loss values of the generators, the pixel-by-pixeldifference loss values, and the total loss value, and minimizing theloss values of the discriminators, the loss values of the generators,the pixel-by-pixel difference loss values and the total loss value toacquire the trained target network.

Specifically, in the S602051, the the loss value LossGAN of thegenerative adversarial network is calculated according to the loss valueLossD_(A) of the first discriminator, the loss value LossD_(H) of thesecond discriminator, the loss value LossD_(V) of the thirddiscriminator and the loss value LossD_(D) of the fourth discriminatorcalculated in step S60205, and the global contour weight α₁, the texturedetail weight α₂, the transversal detail weight β₁, the longitudinaldetail weight β₂, the diagonal detail weight β₃ and the formulaLossGAN=α₁LossD_(A)+α₂(β₁LossD_(H)+β₂LossD_(V)+β₃LossD_(D)). Among them,the global contour weight α₁, the texture detail weight α₂, thetransversal detail weight β₁, the longitudinal detail weight β₂ and thediagonal detail weight β₃ are all hyperparameters acquired throughadjustment in advance according to the peak signal noise ranting (psnr)of the image. In the embodiments of the present application, the globalcontour weight α₁ and the texture detail weight α₂ are introduced tobalance the weight between the global contour feature information andthe texture detail feature information, so as to adjust the globalcontour and detail texture in the reconstructed image; the transversaldetail weight β₁, the longitudinal detail weight β₂ and the diagonaldetail weight β₃ are introduced to adjust the ratio of the transversal,longitudinal and diagonal detail feature information in the image, so asto realize the enhancement processing for the image.

Specifically, in the S602052, the total loss value Loss_(total) iscalculated by the formula Loss_(total)=λ₁LossGAN+λ₂LossF according tothe loss value LossGAN of the generative adversarial network, thepixel-by-pixel difference loss value LossF, the loss weight λ₁ of thegenerative adversarial network, and the pixel-by-pixel difference lossweight λ₂. Among them, the loss weight λ₁ of the generative adversarialnetwork, and the pixel-by-pixel difference loss weight λ₂ arehyperparameters acquired through adjustment in advance according to thepsnr of the image.

Specifically, in the S60206B, when each generative adversarial networkis trained, the network parameters of the generators are fixed first,the loss values of the discriminators are minimized, and the networkparameters of the discriminators are updated through backpropagation tocomplete the training of the discriminators; then the network parametersof the discriminators are fixed, the loss values of the generators areminimized, and the network parameters of the generators are updatedthrough backpropagation; afterward, the pixel-by-pixel difference lossvalue LossF is minimized, and the network parameters of the generatorsand the discriminators are further updated through backpropagation;finally, the total loss value Loss_(total) is minimized, and the networkparameters of the generators and the discriminators are further updatedthrough backpropagation to complete the training of the generativeadversarial network. After the training of each generative adversarialnetwork is completed, the four trained generative adversarial networksare acquired. The four generators of the four generative adversarialnetworks are extracted to acquire the generator group as the trainedtarget network.

In the embodiments of the present application, the total loss value isobtained through the loss values and the weights of the discriminators,so that the network parameters of the generators in the target networkare adjusted through the weights, the generators can generate globalcontour reconstruction information and the detail reconstructioninformation that are provided with a preset proportion, and the contourfeatures and the detail features in the finally obtained reconstructedimage can be accurately presented as expected, thereby improving theaccuracy of the reconstructed image.

In the embodiments of the present application, the four generativeadversarial networks are used to learn the distribution of the overallcontour, transversal detail, longitudinal detail and diagonal texturedetail of the first sample image (having higher resolution) in thewavelet spatial frequency domain, so that each of the generators in thetrained target network can focus on the generation of the global contourfeature information and the detail feature information in each directionof the image, thereby the reconstructed image having higher resolutionand clear details can be synthesized through the inverse wavelettransform based on the accurate and complete feature information, andthe problem that the high-resolution image obtained by converting fromthe low-resolution image is fuzzy is solved.

Further, the image enhancement method in the embodiments of the presentapplication is applied to image enhancement for medical images,correspondingly, the first sample image is high-resolution medical imagesample data.

During clinical diagnosis, medical imaging features of a patient aremain basis for a clinician to make an accurate diagnosis. Therefore,when pathological medical imaging is performed on the patient,acquisition of high-resolution medical images having clearerpathological details helps the clinician perform more accurate analysisand more accurate diagnosis of the patient's condition. However, inbasic medical institutions such as community hospitals, their equipmentonly supports low-resolution general medical imaging, and does notsupport high-resolution medical image scanning. Moreover, theacquisition of high-resolution medical images through CT (ComputedTomography) requires the use of high-dose contrast agent, but the use ofhigh-dose contrast agent brings other potential risks to the patient,such as inducing renal failure and bradycardia etc. Regarding MRI(Magnetic Resonance Imaging), it takes a long scan time (about 30minutes) to obtain high-resolution images, during which the patientneeds to remain still, and it is likely to cause problems such asghosting and blurring of the imaging results if displacement occurs. Atthe same time, long scanning time also causes heavier labor and largertime cost to the doctor, ultimately increasing the medical costs of thepatient. To sum up, in the field of medical imaging, low-resolutionmedical images are usually collected first, and then the collectedlow-resolution medical images are converted into high-resolution medicalimages to assist in medical diagnosis. In order to better assist medicaldiagnosis, it is necessary to solve the problem of how to accuratelyconvert low-resolution medical images into clear high-resolution medicalimages.

In the embodiments of the present application, the steps of theabove-mentioned image enhancement method are specifically applied to theimage enhancement of medical images to solve the above-mentionedproblems. Specifically, the collected low-resolution medical image isused as the image to be processed, and the reconstructed image obtainedthrough the processing of the steps of the image enhancement method isthe high-resolution medical image. Correspondingly, when the targetnetwork is trained based on the four generative adversarial networks,the high-resolution medical image sample data is specifically used asthe first sample image, and the low-resolution medical image sample dataacquired through performing down-sampling on the high-resolution medicalimage sample data is used as the second sample image to train the targetnetwork, so that the final trained target network can accurately learnthe feature information of the high-resolution medical image sampledata, thereby making the reconstructed image obtained by the inversewavelet transform is the accurate high-resolution medical image havingclear details. Further, the embodiments of the present application canalso determine weight parameters such as the global contour weight α₁,the texture detail weight α₂, the weight to balance between the globalcontour feature information and the texture detail feature information,the transversal detail weight Pi, the longitudinal detail weight β₂, thediagonal detail weight β₃ etc. according to the area position of thelesion required to be analyzed in the medical image, so as to enhancethe detail information of the lesion in medical image.

An embodiment of the present application further provides an imageenhancement apparatus, as shown in FIG. 7 , only the parts related tothe embodiment of the present application are shown for ease ofdescription.

This image apparatus includes: a to-be-processed image acquisition unit71, a wavelet transformation unit 72, a reconstruction featureinformation acquisition unit 73, and an inverse wavelet transformationunit 74.

The to-be-processed image acquisition unit 71 is configured to acquirean image to be processed.

The wavelet transform unit 72 is configured to perform a wavelettransform operation on the image to be processed to acquire raw featureinformation of the image to be processed, where the raw featureinformation includes global contour feature information, transversaldetail feature information, longitudinal detail feature information, anddiagonal detail feature information.

Further, the wavelet transform operation is specifically a symmetricalcompactly supported orthogonal wavelet transform operation.

Further, the four generative adversarial networks are specifically fourrelative average generative adversarial networks.

The reconstruction feature information acquisition unit 73 is configuredto input the raw feature information into a trained target network forprocessing to acquire corresponding reconstruction feature information;where the reconstruction feature information includes global contourreconstruction information, transversal detail reconstructioninformation, longitudinal detail reconstruction information and diagonaldetail reconstruction information; the target network is a generatorgroup acquired through training a first sample image and a correspondingsecond sample image based on four generative adversarial networks;resolution of the first sample image is higher than resolution of thesecond sample image.

The inverse wavelet transform unit 74 is configured to perform aninverse wavelet transform operation on the reconstruction featureinformation to acquire a reconstructed image; here resolution of thereconstructed image is higher than resolution of the image to beprocessed.

Further, the wavelet transform unit includes:

a first wavelet transform module configured to perform a wavelettransform operation on the image to be processed in an x-axis directionto acquire first spectrum information;a second wavelet transform module configured to perform a wavelettransform operation on the first spectrum information in a y-axisdirection to acquire second spectrum information;a third wavelet transform module configured to perform a wavelettransform operation on the second spectrum information in a z-axisdirection to acquire third spectrum information;a raw feature information acquisition module configured to acquire theraw feature information according to the third spectrum information.

Further, the trained target network includes a first generator, a secondgenerator, a third generator and a fourth generator, and thereconstruction feature information acquisition unit includes:

a global contour reconstruction information acquisition moduleconfigured to input the global contour feature information into thefirst generator for processing to acquire the global contourreconstruction information;a transversal detail reconstruction information acquisition moduleconfigured to input the transversal detail feature information into thesecond generator for processing to acquire the transversal detailreconstruction information;a longitudinal detail reconstruction information acquisition moduleconfigured to input the longitudinal detail feature information into thethird generator for processing to acquire the longitudinal detailreconstruction information;a diagonal detail reconstruction information acquisition moduleconfigured to input the diagonal detail feature information into thefourth generator for processing to acquire the diagonal detailreconstruction information.

Further, the global contour reconstruction information acquisitionmodule is specifically configured to: input the global contour featureinformation into the first generator and acquire n first intermediatefeature information through n first residual blocks; and acquire theglobal contour reconstruction information according to the n firstintermediate feature information;

the transversal detail reconstruction information acquisition module isspecifically configured to: input the transversal detail featureinformation into the second generator and acquire n second intermediatefeature information through n second residual blocks; and acquire thetransversal detail reconstruction information according to the n secondintermediate feature information;the longitudinal detail reconstruction information acquisition module isspecifically configured to: input the longitudinal detail featureinformation into the third generator, and acquire n third intermediatefeature information through n third residual blocks; and acquire thelongitudinal detail reconstruction information according to the n thirdintermediate feature information;the diagonal detail reconstruction information acquisition module isspecifically configured to: input the diagonal detail featureinformation into the fourth generator and acquire n fourth intermediatefeature information through n fourth residual blocks; and acquire thediagonal detail reconstruction information according to the n fourthintermediate feature information.

Further, the image enhancement apparatus further includes:

a sample image acquisition unit configured to acquire a first sampleimage and perform down-sampling on the first sample image to acquire acorresponding second sample image;a training unit configured to train four generative adversarial networksaccording to the first sample image and the second sample image toacquire the trained target network, where the four generativeadversarial networks include a generator group and a correspondingdiscriminator group, and the target network is the generator group inthe four generative adversarial networks.

Further, the training unit includes:

a first wavelet transform module configured to perform wavelettransformation on the first sample image to acquire first raw featureinformation of the first sample image, where the first raw featureinformation includes first global contour feature information, firsttransversal detail feature information, first longitudinal detailfeature information and first diagonal detail feature information;a second wavelet transform module configured to perform wavelettransform on the second sample image to acquire second raw featureinformation of the second sample image, where the second raw featureinformation includes second global contour feature information, secondtransversal detail feature information, second longitudinal detailfeature information and second diagonal detail feature information;a sample data reconstruction information acquisition module configuredto input the second raw feature information into the generator group forprocessing to acquire corresponding sample data reconstructioninformation, where the sample data reconstruction information includesglobal sample contour reconstruction information, transversal sampledetail reconstruction information, longitudinal sample detailreconstruction information and diagonal sample detail reconstructioninformation;a discrimination module configured to input the first raw featureinformation and the sample data reconstruction information into thediscriminator group for processing to acquire a correspondingdiscrimination output result;a first calculation module configured to calculate loss values of thediscriminators in the discriminator group and loss values of thegenerators in the generator group according to the discrimination outputresult;a training module configured to: iteratively update network parametersof the discriminators and the generators through a gradient descentbackpropagation algorithm respectively according to each of the lossvalues of the discriminators and each of the loss values of thegenerators, and minimize each of the loss values of the discriminatorsand each of the loss values of the generators to acquire the trainedtarget network.

Further, the training unit further includes:

a sample reconstruction image acquisition module configured to performan inverse wavelet transform operation on the sample data reconstructioninformation to acquire a reconstructed sample image;a second calculation module configured to: compare the reconstructedsample image with the corresponding first sample image pixel by pixel,and calculate a pixel-by-pixel difference loss value;correspondingly, the training module is specifically configured to:iteratively update the network parameters of the discriminators and thegenerators through the gradient descent backpropagation algorithmrespectively according to each of the loss values of the discriminators,each of the loss values of the generators and the pixel-by-pixeldifference loss value, and minimize each of the loss values of thediscriminators, each of the loss values of the generators and thepixel-by-pixel difference loss value to acquire the trained targetnetwork.

Further, the training unit further includes:

a third calculation module configured to acquire a loss value of thegenerative adversarial network according to a discrimination loss valuecorresponding to each of the discriminators, a global contour weight, atexture detail weight, a transversal detail weight, a longitudinaldetail weight and a diagonal detail weight;a fourth calculation module configured to calculate a total loss valueaccording to the loss value of the generative adversarial network, thepixel-by-pixel difference loss value, a loss weight of the generativeadversarial network and a pixel-by-pixel difference loss weight;correspondingly, the training module is specifically configured to:iteratively update the network parameters of the discriminators and thediscriminators through the gradient descent backpropagation algorithmrespectively according to each of the loss values of the discriminators,each of the loss values of the generators, the pixel-by-pixel differenceloss value and the total loss value, and minimize each of the lossvalues of the discriminators, each of the loss values of the generators,the pixel-by-pixel difference loss value and the total loss value toacquire the trained target network.

Further, the image enhancement apparatus is applied to image enhancementof three-dimensional medical images, and correspondingly the firstsample image is high-resolution three-dimensional medical image sampledata.

Please refer to FIG. 8 , FIG. 8 is a schematic diagram of a terminaldevice provided by another embodiment of the present application. Asshown in FIG. 8 , the terminal device of this embodiment includes: aprocessor 80, a memory 81, and a computer program 82 stored in thememory 81 and executable on the processor 80. When the processor 80executes the computer program 82, the above-mentioned steps of in theembodiments of the image enhancement method are implemented by theterminal device, for example, S101 to S104 as shown in FIG. 1 .Alternatively, when the processor 80 executes the computer program 82,the functions of the units in the foregoing embodiments are implemented,for example, the functions of the units from 71 to 74 as shown in FIG. 7.

Exemplarily, the computer program 82 may be divided into one or moreunits, and the one or more units are stored in the memory 81 andexecuted by the processor 80 to complete the present application. Theone or more units may be a series of computer program segments capableof completing specific functions, and the program segments are used todescribe the execution process of the computer program 82 in theterminal device 8. For example, the computer program 82 may be dividedinto an acquisition unit, a preprocessing unit and a classificationunit, and the specific functions of the units are as described above.The terminal device may include, but is not limited to, the processor 80and the memory 81. Those skilled in the art should understand that FIG.7 is only an example of the terminal device 8 and does not constitute alimitation on the terminal device 8, which may include more or lesscomponents than those as shown in the figure, or combine certaincomponents, or include different components, for example, the terminaldevice may further include an input and output terminal device, anetwork access terminal device, a bus, and the like.

The so-called processor 80 may be a CPU (Central Processing Unit), andmay also be other general-purpose processor, DSP (Digital SignalProcessor), ASIC (Application Specific Integrated Circuit), FPGA(Field-Programmable Gate Array), or other programmable logic device,discrete gate or transistor logic device, discrete hardware component,etc. The general-purpose processor may be a microprocessor or theprocessor may also be any conventional processor or the like. The memory81 may be an internal storage unit of the terminal device 8, such as ahard disk or storage of the terminal device 8. The memory 81 may also bean external storage terminal device of the terminal device 8, such as aplug-in hard disk, a SMC (Smart Media Card), a SD (Secure Digital) card,a flash card etc. equipped on the terminal device 8. Further, the memory81 may also include both an internal storage unit of the terminal device8 and an external storage terminal device. The memory 81 is used tostore the computer program and other programs and data required by theterminal device. The memory 81 may also be used to temporarily storedata that has been output or will be output. The above-mentionedembodiments are only used to illustrate, but not to limit, the technicalsolutions of the present application; although the present applicationhas been described in detail with reference to the foregoingembodiments, those of ordinary skill in the art should understand that:they can still modify the technical solutions recited in the foregoingembodiments, or equivalently replaces some of the technical featurestherein; and these modifications or replacements do not cause theessence of the corresponding technical solutions to deviate from thespirit and scope of the technical solutions of the embodiments of thepresent application, and should be included within the protection scopeof the present application.

1. An image enhancement method, comprising: acquiring an image to beprocessed; performing a wavelet transform operation on the image to beprocessed to acquire raw feature information of the image to beprocessed, wherein the raw feature information includes global contourfeature information, transversal detail feature information,longitudinal detail feature information, and diagonal detail featureinformation; inputting the raw feature information into a trained targetnetwork for processing to acquire corresponding reconstruction featureinformation; wherein the reconstruction feature information includesglobal contour reconstruction information, transversal detailreconstruction information, longitudinal detail reconstructioninformation and diagonal detail reconstruction information, the targetnetwork is a generator group acquired through training a first sampleimage and a corresponding second sample image based on four generativeadversarial networks, and resolution of the first sample image is higherthan resolution of the second sample image; performing an inversewavelet transform operation on the reconstruction feature information toacquire a reconstructed image; wherein resolution of the reconstructedimage is higher than resolution of the image to be processed.
 2. Theimage enhancement method of claim 1, wherein the wavelet transformoperation is specifically a symmetrical compactly supported orthogonalwavelet transform operation.
 3. The image enhancement method of claim 1,wherein the four generative adversarial networks are specifically fourrelative mean generative adversarial networks.
 4. The image enhancementmethod of claim 1, wherein the image to be processed is specifically athree-dimensional image, and the performing a wavelet transformoperation on the image to be processed to acquire raw featureinformation of the image to be processed comprises: performing a wavelettransform operation on the image to be processed in an x-axis directionto acquire first spectrum information; performing a wavelet transformoperation on the first spectrum information in a y-axis direction toacquire second spectrum information; performing a wavelet transformoperation on the second spectrum information in a z-axis direction toacquire third spectrum information; acquiring the raw featureinformation according to the third spectrum information.
 5. The imageenhancement method of claim 1, wherein the trained target networkcomprises a first generator, a second generator, a third generator and afourth generator, and the inputting the raw feature information into atrained target network for processing to acquire correspondingreconstruction feature information comprises: inputting the globalcontour feature information into the first generator for processing toacquire the global contour reconstruction information; inputting thetransversal detail feature information into the second generator forprocessing to acquire the transversal detail reconstruction information;inputting the longitudinal detail feature information into the thirdgenerator for processing to acquire the longitudinal detailreconstruction information; inputting the diagonal detail featureinformation into the fourth generator for processing to acquire thediagonal detail reconstruction information.
 6. The image enhancementmethod of claim 5, wherein the first generator, the second generator,the third generator and the fourth generator are all residual networkstructures including n residual blocks, wherein n is a positive integer;correspondingly, the inputting the raw feature information into atrained target network for processing to acquire correspondingreconstruction feature information comprises: inputting the globalcontour feature information into the first generator and acquiring nfirst intermediate feature information through n first residual blocks;and acquiring the global contour reconstruction information according tothe n first intermediate feature information; inputting the transversaldetail feature information into the second generator and acquiring nsecond intermediate feature information through n second residualblocks; and acquire the transversal detail reconstruction informationaccording to the n second intermediate feature information; inputtingthe longitudinal detail feature information into the third generator,and acquiring n third intermediate feature information through n thirdresidual blocks; and acquiring the longitudinal detail reconstructioninformation according to the n third intermediate feature information;inputting the diagonal detail feature information into the fourthgenerator and acquiring n fourth intermediate feature informationthrough n fourth residual blocks; and acquiring the diagonal detailreconstruction information according to the n fourth intermediatefeature information.
 7. The image enhancement method of claim 1,wherein, before acquiring the image to be processed, further comprising:acquiring a first sample image, and performing down-sampling on thefirst sample image to acquire a corresponding second sample image;training four generative adversarial networks according to the firstsample image and the second sample image to acquire the trained targetnetwork, wherein the four generative adversarial networks include agenerator group and a corresponding discriminator group, and the targetnetwork is the generator group in the four generative adversarialnetworks.
 8. The image enhancement method of claim 7, wherein thetraining four generative adversarial networks according to the firstsample image and the second sample image to acquire the trained targetnetwork comprises: performing wavelet transformation on the first sampleimage to acquire first raw feature information of the first sampleimage, wherein the first raw feature information comprises first globalcontour feature information, first transversal detail featureinformation, first longitudinal detail feature information and firstdiagonal detail feature information; performing wavelet transform on thesecond sample image to acquire second raw feature information of thesecond sample image, wherein the second raw feature informationcomprises second global contour feature information, second transversaldetail feature information, second longitudinal detail featureinformation and second diagonal detail feature information; inputtingthe second raw feature information into the generator group forprocessing to acquire corresponding sample data reconstructioninformation, wherein the sample data reconstruction informationcomprises global sample contour reconstruction information, transversalsample detail reconstruction information, longitudinal sample detailreconstruction information and diagonal sample detail reconstructioninformation; inputting the first raw feature information and the sampledata reconstruction information into the discriminator group forprocessing to acquire a corresponding discrimination output result;calculating loss values of discriminators in the discriminator group andloss values of generators in the generator group according to thediscrimination output result; iteratively updating network parameters ofthe discriminators and the generators through a gradient descentbackpropagation algorithm respectively according to each of the lossvalues of the discriminators and each of the loss values of thegenerators, and minimizing each of the loss values of the discriminatorsand each of the loss values of the generators to acquire the trainedtarget network.
 9. The image enhancement method of claim 8, wherein,after inputting the second raw feature information into the generatorgroup for processing to acquire corresponding sample data reconstructioninformation, further comprising: performing an inverse wavelet transformoperation on the sample data reconstruction information to acquire areconstructed sample image; comparing the reconstructed sample imagewith the corresponding first sample image pixel by pixel, andcalculating a pixel-by-pixel difference loss value; correspondingly, theiteratively updating network parameters of the discriminators and thegenerators through a gradient descent backpropagation algorithmrespectively according to each of the loss values of the discriminatorsand each of the loss values of the generators and minimizing each of theloss values of the discriminators and each of the loss values of thegenerators to acquire the trained target network comprise: iterativelyupdate the network parameters of the discriminators and the generatorsthrough the gradient descent backpropagation algorithm respectivelyaccording to each of the loss values of the discriminators, each of theloss values of the generators and the pixel-by-pixel difference lossvalue, and minimizing each of the loss values of the discriminators,each of the loss values of the generators and the pixel-by-pixeldifference loss value to acquire the trained target network.
 10. Theimage enhancement method of claim 9, wherein, after calculating lossvalues of the discriminators in the discriminator group and loss valuesof the generators in the generator group according to the discriminationoutput result, further comprising: acquiring a loss value of thegenerative adversarial network according to a discrimination loss valuecorresponding to each of the discriminators, a global contour weight, atexture detail weight, a transversal detail weight, a longitudinaldetail weight and a diagonal detail weight; calculating a total lossvalue according to the loss value of the generative adversarial network,the pixel-by-pixel difference loss value, a loss weight of thegenerative adversarial network and a pixel-by-pixel difference lossweight; correspondingly, the iteratively updating network parameters ofthe discriminators and the generators through a gradient descentbackpropagation algorithm respectively according to each of the lossvalues of the discriminators and each of the loss values of thegenerators and minimizing each of the loss values of the discriminatorsand each of the loss values of the generators to acquire the trainedtarget network comprise: iteratively updating the network parameters ofthe discriminators and the discriminators through the gradient descentbackpropagation algorithm respectively according to each of the lossvalues of the discriminators, each of the loss values of the generators,the pixel-by-pixel difference loss value and the total loss value, andminimizing each of the loss values of the discriminators, each of theloss values of the generators, the pixel-by-pixel difference loss valueand the total loss value to acquire the trained target network.
 11. Theimage enhancement method of claim 7, wherein the image enhancementmethod is applied to image enhancement of medical images, andcorrespondingly the first sample image is high-resolution medical imagesample data. 12.-18. (canceled)
 19. A terminal device, comprising amemory, a processor and a computer program stored in the memory andexecutable on the processor, wherein the processor, when executing thecomputer program, causes the terminal device to implement steps of:acquiring an image to be processed; performing a wavelet transformoperation on the image to be processed to acquire raw featureinformation of the image to be processed, wherein the raw featureinformation includes global contour feature information, transversaldetail feature information, longitudinal detail feature information, anddiagonal detail feature information; inputting the raw featureinformation into a trained target network for processing to acquirecorresponding reconstruction feature information; wherein thereconstruction feature information includes global contourreconstruction information, transversal detail reconstructioninformation, longitudinal detail reconstruction information and diagonaldetail reconstruction information, the target network is a generatorgroup acquired through training a first sample image and a correspondingsecond sample image based on four generative adversarial networks, andresolution of the first sample image is higher than resolution of thesecond sample image; performing an inverse wavelet transform operationon the reconstruction feature information to acquire a reconstructedimage; wherein resolution of the reconstructed image is higher thanresolution of the image to be processed.
 20. A computer-readable storagemedium storing a computer program, wherein the computer program, whenexecuted by a processor, causes a terminal device to implement steps of:acquiring an image to be processed; performing a wavelet transformoperation on the image to be processed to acquire raw featureinformation of the image to be processed, wherein the raw featureinformation includes global contour feature information, transversaldetail feature information, longitudinal detail feature information, anddiagonal detail feature information; inputting the raw featureinformation into a trained target network for processing to acquirecorresponding reconstruction feature information; wherein thereconstruction feature information includes global contourreconstruction information, transversal detail reconstructioninformation, longitudinal detail reconstruction information and diagonaldetail reconstruction information, the target network is a generatorgroup acquired through training a first sample image and a correspondingsecond sample image based on four generative adversarial networks, andresolution of the first sample image is higher than resolution of thesecond sample image; performing an inverse wavelet transform operationon the reconstruction feature information to acquire a reconstructedimage; wherein resolution of the reconstructed image is higher thanresolution of the image to be processed.
 21. The terminal device ofclaim 19, wherein the image to be processed is specifically athree-dimensional image, and the performing a wavelet transformoperation on the image to be processed to acquire raw featureinformation of the image to be processed comprises: performing a wavelettransform operation on the image to be processed in an x-axis directionto acquire first spectrum information; performing a wavelet transformoperation on the first spectrum information in a y-axis direction toacquire second spectrum information; performing a wavelet transformoperation on the second spectrum information in a z-axis direction toacquire third spectrum information; acquiring the raw featureinformation according to the third spectrum information.
 22. Theterminal device of claim 19, wherein the trained target networkcomprises a first generator, a second generator, a third generator and afourth generator, and the inputting the raw feature information into atrained target network for processing to acquire correspondingreconstruction feature information comprises: inputting the globalcontour feature information into the first generator for processing toacquire the global contour reconstruction information; inputting thetransversal detail feature information into the second generator forprocessing to acquire the transversal detail reconstruction information;inputting the longitudinal detail feature information into the thirdgenerator for processing to acquire the longitudinal detailreconstruction information; inputting the diagonal detail featureinformation into the fourth generator for processing to acquire thediagonal detail reconstruction information.
 23. The terminal device ofclaim 22, wherein the first generator, the second generator, the thirdgenerator and the fourth generator are all residual network structuresincluding n residual blocks, wherein n is a positive integer;correspondingly, the inputting the raw feature information into atrained target network for processing to acquire correspondingreconstruction feature information comprises: inputting the globalcontour feature information into the first generator and acquiring nfirst intermediate feature information through n first residual blocks;and acquiring the global contour reconstruction information according tothe n first intermediate feature information; inputting the transversaldetail feature information into the second generator and acquiring nsecond intermediate feature information through n second residualblocks; and acquire the transversal detail reconstruction informationaccording to the n second intermediate feature information; inputtingthe longitudinal detail feature information into the third generator,and acquiring n third intermediate feature information through n thirdresidual blocks; and acquiring the longitudinal detail reconstructioninformation according to the n third intermediate feature information;inputting the diagonal detail feature information into the fourthgenerator and acquiring n fourth intermediate feature informationthrough n fourth residual blocks; and acquiring the diagonal detailreconstruction information according to the n fourth intermediatefeature information.
 24. The terminal device of claim 19, wherein beforeacquiring the image to be processed, the processor, when executing thecomputer program, causes the terminal device to further implement stepsof: acquiring a first sample image, and performing down-sampling on thefirst sample image to acquire a corresponding second sample image;training four generative adversarial networks according to the firstsample image and the second sample image to acquire the trained targetnetwork, wherein the four generative adversarial networks include agenerator group and a corresponding discriminator group, and the targetnetwork is the generator group in the four generative adversarialnetworks.
 25. The terminal device of claim 24, wherein the training fourgenerative adversarial networks according to the first sample image andthe second sample image to acquire the trained target network comprises:performing wavelet transformation on the first sample image to acquirefirst raw feature information of the first sample image, wherein thefirst raw feature information comprises first global contour featureinformation, first transversal detail feature information, firstlongitudinal detail feature information and first diagonal detailfeature information; performing wavelet transform on the second sampleimage to acquire second raw feature information of the second sampleimage, wherein the second raw feature information comprises secondglobal contour feature information, second transversal detail featureinformation, second longitudinal detail feature information and seconddiagonal detail feature information; inputting the second raw featureinformation into the generator group for processing to acquirecorresponding sample data reconstruction information, wherein the sampledata reconstruction information comprises global sample contourreconstruction information, transversal sample detail reconstructioninformation, longitudinal sample detail reconstruction information anddiagonal sample detail reconstruction information; inputting the firstraw feature information and the sample data reconstruction informationinto the discriminator group for processing to acquire a correspondingdiscrimination output result; calculating loss values of discriminatorsin the discriminator group and loss values of generators in thegenerator group according to the discrimination output result;iteratively updating network parameters of the discriminators and thegenerators through a gradient descent backpropagation algorithmrespectively according to each of the loss values of the discriminatorsand each of the loss values of the generators, and minimizing each ofthe loss values of the discriminators and each of the loss values of thegenerators to acquire the trained target network.
 26. The terminaldevice of claim 25, wherein after inputting the second raw featureinformation into the generator group for processing to acquirecorresponding sample data reconstruction information, the processor,when executing the computer program, causes the terminal device tofurther implement steps of: performing an inverse wavelet transformoperation on the sample data reconstruction information to acquire areconstructed sample image; comparing the reconstructed sample imagewith the corresponding first sample image pixel by pixel, andcalculating a pixel-by-pixel difference loss value; correspondingly, theiteratively updating network parameters of the discriminators and thegenerators through a gradient descent backpropagation algorithmrespectively according to each of the loss values of the discriminatorsand each of the loss values of the generators and minimizing each of theloss values of the discriminators and each of the loss values of thegenerators to acquire the trained target network comprise: iterativelyupdate the network parameters of the discriminators and the generatorsthrough the gradient descent backpropagation algorithm respectivelyaccording to each of the loss values of the discriminators, each of theloss values of the generators and the pixel-by-pixel difference lossvalue, and minimizing each of the loss values of the discriminators,each of the loss values of the generators and the pixel-by-pixeldifference loss value to acquire the trained target network.
 27. Theterminal device of claim 26, wherein after calculating loss values ofthe discriminators in the discriminator group and loss values of thegenerators in the generator group according to the discrimination outputresult, the processor, when executing the computer program, causes theterminal device to further implement steps of: acquiring a loss value ofthe generative adversarial network according to a discrimination lossvalue corresponding to each of the discriminators, a global contourweight, a texture detail weight, a transversal detail weight, alongitudinal detail weight and a diagonal detail weight; calculating atotal loss value according to the loss value of the generativeadversarial network, the pixel-by-pixel difference loss value, a lossweight of the generative adversarial network and a pixel-by-pixeldifference loss weight; correspondingly, the iteratively updatingnetwork parameters of the discriminators and the generators through agradient descent backpropagation algorithm respectively according toeach of the loss values of the discriminators and each of the lossvalues of the generators and minimizing each of the loss values of thediscriminators and each of the loss values of the generators to acquirethe trained target network comprise: iteratively updating the networkparameters of the discriminators and the discriminators through thegradient descent backpropagation algorithm respectively according toeach of the loss values of the discriminators, each of the loss valuesof the generators, the pixel-by-pixel difference loss value and thetotal loss value, and minimizing each of the loss values of thediscriminators, each of the loss values of the generators, thepixel-by-pixel difference loss value and the total loss value to acquirethe trained target network.